Symmetric Publish / Subscribe via Constraint Publication

Traditional publish / subscribe systems offer a range of expressive subscription languages for constraints, but restrict the publish operation to be a single published object that contains only constants and no constraints. We have introduced a novel generalization of publish / subscribe called symmetric publish / subscribe where both publications and subscriptions contain constraints in addition to constants, and published objects are matched to subscriptions by computing the intersection of their constraints. This paper describes the challenge of empirically evaluating symmetric publish / subscribe, a system for which traditional publish / subscribe benchmarking and evaluation tools do not completely apply. We introduce a number of workloads designed to elucidate the performance issues in our implementation, discuss the advantages and disadvantages of our evaluation methodology, and discuss how our experience generalizes to the evaluation of other novel publish / subscribe and database systems for which no established evaluation methodology exists.